Time-Of-Flight camera system

ABSTRACT

The invention relates to a TOF camera system comprising several cameras, at least one of the cameras being a TOF camera, wherein the cameras are assembled on a common substrate and are imaging the same scene simultaneously and wherein at least two cameras are driven by different driving parameters.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit under 35 U.S.C. § 120 as acontinuation application of U.S. application Ser. No. 16/460,049, filedon Jul. 2, 2019, now U.S. Pat. No. 10,638,118, which is a division ofU.S. patent application Ser. No. 14/904,554, filed on Jan. 12, 2016, nowU.S. Pat. No. 10,397,552, which claims the benefit under 35 U.S.C. § 371as a U.S. National Stage Entry of International Application No.PCT/EP2014/079304, filed in the European Patent Office as a ReceivingOffice on Dec. 24, 2014, which claims priority to European PatentApplication No. 13199564.9, filed in the European Patent Office on Dec.24, 2013, each of which is hereby incorporated by reference in itsentirety.

FIELD OF THE INVENTION

The present invention relates to Time-Of-Flight (TOF) range imagingsystems, namely TOF camera systems. In particular, the aim of thepresent invention is to provide a 3D image of a scene of high quality.

BACKGROUND OF THE INVENTION

Computer vision is a growing research field that includes methods foracquiring, processing, analysing, and understanding images. The maindriving idea in that field is to duplicate the abilities of the humanvision system by electronically perceiving and understanding images of ascene. Notably, one theme of research in computer vision is the depthperception or, in other words, the three-dimensional (3D) vision.

For human beings, the depth perception is originated from the so-calledstereoscopic effect by which the human brain fuses two slightlydifferent images of a scene captured by the two eyes, and retrieves,among others, depth information. Moreover, recent studies have shownthat the capacity to recognize objects in a scene greatly furthercontributes to the depth perception.

For camera systems, the depth information is not easily obtained andrequires complex methods and systems. When imaging a scene, oneconventional two-dimensional (2D) camera system associates each point ofthe scene with a given RGB colour information. At the end of the imagingprocess, a 2D colour map of the scene is created. A standard 2D camerasystem cannot recognize objects in a scene easily from that colour mapas colour is highly dependent on varying scene illumination and as itdoes not intrinsically contain any dimensional information. Newtechnologies have been introduced for developing computer vision andnotably for developing 3D imaging, enabling in particular the directcapture of depth related information and the indirect acquisition ofscene or object related dimensional information. The recent advancementsin 3D imaging systems are impressive and have led to a growing interestfrom industry, academy and consumer society.

The most common technologies used to create 3D images are based on thestereoscopic effect. Two cameras take pictures of the same scene, butthey are separated by a distance—exactly like the human eyes. A computercompares the images while shifting the two images together over top ofeach other to find the parts that match and those that mismatch. Theshifted amount is called the disparity. The disparity at which objectsin the image best match is used by the computer to calculate distanceinformation, namely a depthmap, by using additionally camera sensorsgeometrical parameters and lens specifications.

Another more recent and different technology is represented by theTime-Of-Flight (TOF) camera system 3 illustrated in FIG. 1. TOF camerasystem 3 includes a camera 1 with a dedicated illumination unit 18 anddata processing means 4. TOF camera systems capable of capturing 3Dimages of a scene 15 by analysing the time of flight of light from alight source 18 to an object. Such 3D camera systems are now used inmany applications where depth or distance information measurement isrequired. Standard 2D camera systems, such as Red-Green-Blue (RGB)camera systems, are passive technologies, i.e. they use the ambientlight to capture images and are not based on the emission of anadditional light. On the contrary, the basic operational principle of aTOF camera system is to actively illuminate a scene 15 with a modulatedlight 16 at a predetermined wavelength using the dedicated illuminationunit, for instance with some light pulses of at least one predeterminedfrequency. The modulated light is reflected back from objects within thescene. A lens collects the reflected light 17 and forms an image of theobjects onto an imaging sensor 1. Depending on the distance of objectsfrom the camera, a delay is experienced between the emission of themodulated light, e.g. the so called light pulses, and the reception atthe camera of those light pulses. In one common embodiment, distance inbetween reflecting objects and the camera may be determined as functionof the time delay observed and the speed of light constant value. In oneanother more complex and reliable embodiment, a plurality of phasedifference in between the emitted reference light pulses and thecaptured light pulses may be determined and used for estimating depthinformation as introduced in Robert Lange phd thesis entitled “3Dtime-of-flight distance measurement with custom solid-state imagesensors in CMOS/CCD technology”.

A TOF camera system comprises several elements, each of them having adistinct function.

1) A first component of a TOF camera system is the illumination unit 18.When using pulses, the pulse width of each light pulse determines thecamera range. For instance, for a pulse width of 50 ns, the range islimited to 7.5 m. As a consequence, the illumination of the scenebecomes critical to the operation of a TOF camera system, and the highspeed driving frequency requirements for illumination units necessitatethe use of specialised light sources such as light emitting diodes(LEDs) or lasers to generate such short light pulses.2) Another component of a TOF camera system is the imaging sensor 1 orTOF sensor. The imaging sensor typically comprises a matrix array ofpixels forming an image of the scene. By pixel, it should be understoodthe picture element sensitive to light electromagnetic radiations aswell as its associated electronic circuitry. The output of the pixelscan be used to determine the time of flight of light from theillumination unit to an object in the scene and reflected back from theobject to the imaging TOF sensor. The time of flight can be calculatedin a separate processing unit which may be coupled to the TOF sensor ormay directly be integrated into the TOF sensor itself. Various methodsare known for measuring the timing of the light as it travels from theillumination unit to the object and from the object back to the imagingsensor.3) Imaging optics 2 and processing electronics 4 are also providedwithin a TOF camera system. The imaging optics are designed to gatherthe reflected light from objects in the scene, usually in the IR domain,and filter out light that is not in the same wavelength than the lightemitted by the illumination unit. In some embodiments, the optics mayenable the capture of infra-red illumination for TOF principlemeasurements and visible illumination for RGB colour measurements. Theprocessing electronics drives the TOF sensor so as to, among severalfeatures, filter out light of frequencies different from the onesemitted by the illumination unit but having a similar wavelength(typically the sunlight). By filtering out unwanted wavelengths orfrequencies, background light can effectively be suppressed. Theprocessing electronics further include drivers for both the illuminationunit and the imaging sensor so that these components can accurately becontrolled in synchrony to ensure that an accurate image capture isperformed and that a reliable depthmap of the scene is determined.

The choice of elements constituting a TOF camera system is crucial. TOFcamera systems tend to cover wide ranges from a few millimetres up toseveral kilometres depending on the type and on the performances of theelements used. Such TOF camera systems may have distance accuracyvarying from the sub-centimetres to several centimetres or even metres.Technologies that can be used with TOF camera systems include pulsedlight sources with digital time counters, radio frequency (RF) modulatedlight sources with phase detectors, and range-gated imagers.

TOF camera systems suffer from several drawbacks. In current TOF imagersor TOF sensors, pixel pitches are usually ranging from 10 μm to 100 μm.Due to the novelty of the technology and to the fact that thearchitecture of a TOF pixel is highly complex, it is difficult to designa small pixel size while maintaining an efficient signal to noise ratio(SNR) and keeping in mind the requirement related to mass production atlow cost. This results in relatively big chip sizes for TOF imagesensor. With conventional optics, such big sizes of image sensor requirelarge and thick optical stacks to fit onto the die. Generally, acompromise has to be found between required resolution and the thicknessof the device to make it be embeddable on portable mass consumerproduct.

Furthermore, the depth measurement obtained by a TOF camera system maybe erroneously determined for several reasons. Firstly, the resolutionof such systems is to be improved. Big pixel size requires big sensorchip and thus the sensor resolution is limited by the TOF sensor size.Secondly, the accuracy in depth measurement of such systems still needsto be improved as, among a plurality of parameters, it is highlydependent on the Signal to Noise ratio and on the modulation frequency(the modulation frequency determining the depth accuracy and theoperating depth measurement range). In particular, the uncertainty orinaccuracy in depth measurement may be due to an effect called “depthaliasing” which will be described in details later. Moreover,uncertainty can originate from the presence of additional light in thebackground. Indeed, the pixels of TOF camera systems comprise aphotosensitive element which receives incident light and converts itinto an electrical signal, for example, a current signal. During thecapture of a scene, if the background light is too intense in thewavelength the sensor is sensitive to, then pixels may receiveadditional light not reflected from objects within the scene, which mayalter the measured distance.

At present, in the field of TOF imaging, several options are availableto overcome at least partially the major individual drawbacks thetechnology may suffer from, such as for instance, improved modulationfrequency systems enabling more robust and accurate depth measurement,dealiasing or background light robustness mechanisms.

A solution remains to be proposed in to address these drawbacks togetherand to additionally improve the resolution of the TOF camera systemswhile limiting the thickness of the complete system and reducingparallax issues to make it compliant with mass-produced portable devicesintegration.

SUMMARY OF THE INVENTION

The present invention relates to a TOF camera system comprising severalcameras, at least one of the cameras being a TOF camera, wherein thecameras are assembled on a common substrate and are imaging the samescene simultaneously and wherein at least two cameras are driven bydifferent driving parameters.

Using the at least one TOF camera depth information combined with atleast the information from another camera driven with differentparameters, the fusion of all the camera information together helps inrefining and enhancing the quality of the resulting image, and inparticular helps in obtaining a higher quality depthmap from thecaptured scene, since the images are acquired simultaneously by thecameras.

Advantageously, the sensors of the cameras are manufactured andassembled on a common substrate, such as for instance a silicon basedsubstrate or a wafer, which reduces the thickness and the size of theTOF camera system. This common substrate enables also to reduce parallaxissues resulting from the use of several cameras.

Preferably, the TOF camera system also further comprises an array ofseveral lenses, each lens of the array being associated to each of thecameras. These lenses help focusing the impinging light onphotosensitive area of their respective associated camera sensor.

Advantageously, the driving parameters comprise parameters forimplementing a stereoscopic technique and/or for implementing adealiasing algorithm and/or for implementing a background lightrobustness mechanism. Dealiasing shall be explained herein below.

More advantageously, at least two cameras of the TOF camera system mayimage the same scene during different integration times.

More advantageously, the TOF camera system may comprises two TOF camerashaving each a TOF sensor imaging the same scene and being driven fordetermining distance information from different modulation frequencies.

More preferably, the TOF camera system further may comprise means forfiltering the light in the visible range and/or in the InfraRed. The useof such means for filtering the light enables the tuning of the light inorder to choose wavelength in the range of which each sensor have to besensitive to.

The present invention shall be better understood upon reading thefollowing description, in light of the attached drawings.

DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates basic operational principle of a TOF camera system.

FIG. 2 illustrates a multi-lens TOF sensor stack.

FIG. 3 illustrates a standard TOF sensor used in a stack such asillustrated in FIG. 2.

FIG. 4 illustrates a custom optimized TOF sensor for a stack such asillustrated in FIG. 2.

FIG. 5 illustrates a stack, such as illustrated in FIG. 2, using 4separate TOF sensors.

FIG. 6 illustrates a multi-lens TOF sensor stack, also using colour andinfrared filters.

DESCRIPTION OF THE INVENTION

The present invention will be described with respect to particularembodiments and with reference to certain drawings but the invention isnot limited thereto. The drawings are only schematic and arenon-limiting. In the drawings, the size of some of the elements may beexaggerated and not drawn on scale for illustrative purposes.

As illustrated by FIG. 1, a conventional TOF camera system comprises oneTOF sensor 1 and its associated optical means 2 (e.g. a lens), anillumination unit 18 for illuminating the scene 15 with respect to theTOF principle specifications, and an electronic circuitry 4 for at leastdriving the illumination unit and the TOF sensor. The light is usuallyin the infra-red wavelength domain and comprises periodically modulatedpulses 16 emitted toward the scene. The TOF sensor and its associatedoptical means are designed to enable the capture of the emittedmodulated light that is reflected back from the scene. One option fordetermining distance information in-between the scene objects and the soformed TOF camera system is to determine the phase delay between theemitted pulsed or modulated light and the light received back at the TOFsensor.

In order to improve the quality and resolution of a Time-Of-Flightimage, namely the depthmap, and to reduce the thickness of TOF camerasystem, the present invention relates to a novel TOF camera systemcomprising several cameras, at least one of the cameras being a TOFcamera, wherein the cameras are assembled on a common support and areimaging the same scene and wherein at least two cameras are driven bydifferent driving parameters.

By camera, it is meant an electronic device system comprising at leastthe means for capturing the electromagnetic radiation of an impinginglight. For instance, a camera may be represented at least by one singlepixel of a sensor device. A camera may also be represented by a group ofpixels on a sensor device or by an entire sensor device. Preferably, thesensor device from which at least one camera is determined comprises amatrix array of pixels and the circuitry for operating them. Thecircuitry may further comprises electronic means for further processingthe data measured by each pixel and/or each camera from the at least onesensor device used. The invention may also relate more generally to aTOF camera system comprising a plurality of independent camera havingeach at least one sensor device, and among which at least one comprisesa TOF sensor device.

The invention will be now explained with respect to a symmetricconfiguration of a 4-cameras array. It is worth noticing at this pointthat aspects of the present invention are neither limited to fourcameras associated each with at least one lens, nor to the symmetryshown in the used examples. A person skilled in the art could easilyextrapolate the described principles to less, or to more lenses andcameras, for instance two lenses associated to at least one sensor ontowhich two cameras are defined, and/or differently configured viewpoints.

When designing a TOF camera system comprising several cameras, at leastone of the cameras being a TOF camera, several configurations arepossible to arrange the cameras.

In FIG. 2, a first configuration is shown with 4 lenses A, B, C, D(101-104) on top of a support, an image sensor plane 100. Each lensenables the impinging light coming from the imaged scene to be focusedon each individual camera of the image sensor plane. For instance, inone embodiment each lens focuses the captured light on each cameradefined on a TOF image sensor. The fusion of the four individual imagesmay offer a higher resolution image with a lower thickness than a largerhigh resolution single camera TOF sensor system.

In FIG. 3 to FIG. 5, a support, i.e. an image sensor plane 100, fourcameras 107 and their associated circuitry 110 are shown. Severalpossible configurations of the image sensor circuitry within the supportare displayed.

1) The first configuration, illustrated in FIG. 3, is the moststraightforward. One single TOF image sensor device is used; it coversthe four image areas 107 (i.e. the cameras) constructed or delimited bythe four lenses 101-104. The image sensor circuitry 110, comprisingvarious analog and/or digital blocks (signal conditioning,Analog-to-Digital Conversion, filtering, image sensor processing . . .), is in this case shown on the side of the image sensor and all the TOFpixels are grouped. An advantage of this approach is that existing TOFimage sensors devices can be used for this principle. One disadvantageof this approach is that a lot of TOF pixels in-between the regions 107are not in the image plane of the optics 101-104 and are by the wayuseless. Another disadvantage of this approach is that such a systemwill suffer from a limited resolution since an efficient TOF sensordevice is natively limited in resolution for a given size. Anotherdisadvantage of this approach is that it provides only TOF principlebased information from the scene i.e. a depthmap and an illumination orconfidence greyscale map.2) A second possible configuration is shown in FIG. 4, where severalcameras are assembled on a common support (e.g. designed on the samesilicon substrate). In this configuration, each camera is also coveredby its own lens. Only cameras located in the regions delimitated byoptics are generating the images. This way, the image sensor circuitrycan be allocated in the free space between the regions 107. In FIG. 4,the free space between the regions 107 can be seen as rectangularstrips, forming a “cross”, and wherein the electronic circuitry foroperating the cameras can be set so as to save silicon and minimize thesize of the so formed sensor system. As shown in FIG. 4, the imagesensor system obtained is smaller in size than the image sensor systemfrom FIG. 2. This second configuration optimizes cost and board space.It is to be noted that obviously, the electronic circuitry filling thefree substrate space available in between the cameras may be designed inother less optimal forms than a cross, for instance in the form of astripe.3) A third possible configuration is shown in FIG. 5, where four cameras(formed by four individual TOF image sensors) are positioned under thefour lenses 101-104 of FIG. 2 and form one single support together. Inthis configuration, each TOF sensor is covered by its own lens, and isgoverned by its own circuitry. With this approach, four individualcamera calibrations and mounting alignment steps are required.

According to a first embodiment of the present invention, the TOF camerasystem comprises several cameras, at least one of the cameras being aTOF camera, wherein the cameras are assembled on a common substrate andare imaging the same scene simultaneously and wherein at least twocameras are driven by different driving parameters.

By common substrate, it should be understood that the cameras aremanufactured on a common base, i.e. an underlying material providing asurface on which the cameras can directly be manufactured, for instancea wafer such as the ones commonly used in the field of microelectronics.This substrate can be silicon based for instance and the plurality ofcameras can be made from this silicon.

The fact that the cameras are imaging the same scene simultaneouslymeans that the cameras are exposed to the light coming from the scene atthe same time, and not sequentially, in order to obtain an improvedmeasurement demonstrating for instance no motion related artefacts fromone camera capture with some determined parameters to the at least otherone camera capture determined with some other parameters.

The TOF camera system may be designed according to the configurationsexposed above. Preferably, the TOF camera system may be designedaccording the configuration displayed in FIG. 4 wherein the cameras areassembled on a common substrate. This substrate may be silicon-based,but the present invention is not limited thereto.

The facts that the cameras are assembled on a common substrate and areimaging the same scene and that at least two cameras are driven bydifferent driving parameters simultaneously enable in particular toobtain different types of information from the same scenesimultaneously, this information being for example at least one ofcolour, illumination or depthmap information. Preferably, thisinformation may be several depthmaps of a determined resolution andoptionally a colour image of preferably a higher resolution.

The fusion of the different information contained in each single image,namely the fusion of at least one depthmap obtained according to the TOFprinciple with at least another image containing at least depthinformation or colour information, enables the computation of one singleresulting image with improved quality. By “fusion”, it should beunderstood the combination of information related to individual imagesto generate the improved and/or refined resulting image or “super-image”demonstrating at least a higher quality depth measurement for eachsingle pixel or a higher resolution.

By using this TOF camera system, it is possible to fuse individualimages to one “super-image”, for instance to fuse 4 individual images.In one preferred embodiment, both the resolution and the depthmapaccuracy information of the so-called “super-image” resulting from thefusion are improved compared to the individual information generatedfrom each of the single individual images.

In one embodiment, at least one of the lenses of the lens array or atleast one of the cameras of the TOF system may be different from theothers in that, the lens may deliver an image with a different focallength, and the cameras may be of a different size and/or a differentresolution. For instance, a TOF camera system comprising two TOF camerasand two colour camera may have colour cameras (respectively coloursensors) different in size and resolution from the TOF cameras(respectively TOF sensors). The lens associated with the TOF camera mayfurther be of a different focal length than those associated with thecolour cameras. The scene observed by the TOF cameras and the colourcameras being the same, the parameters associated to each kind ofcameras, namely the resolution, the lens focal length, the sensor sizes,may lead to different images captured by each kind of camera. Forinstance a depthmap estimated by stereovision principle from the colourimages may represent a slightly different view of the scene imaged bythe depthmap obtained by at least one TOF camera.

The driving parameters that may be implemented in the TOF camera systemare presented herein below, but are not limited thereto.

In one embodiment, at least two of the cameras may be driven byparameters for implementing a stereoscopic technique. Stereoscopy refersto a technique for creating or enhancing the illusion of depth in animage, by means of binocular vision. In this technique, binocular visionof a scene creates two slightly different images of the scene in the twoeyes, due to the different positions of eyes on the head. Thesedifferences provide information that the brain can use to calculatedepth in the visual scene, providing a depth perception. In oneembodiment, a passive stereoscopic calculation may be used next to thetime-of-flight depth calculation, based on the combinations of at leasttwo viewpoints of the present invention. This calculation may be verycoarse, to identify or resolve dealiasing. Preferably, the furthestapart regions 107 i.e. the furthest cameras may be used. Furtherpreferably, in the case of four pixels, the diagonal regions may be usedto implement those driving parameters.

In one derived embodiment, at least two colour cameras of sameresolution may be used for providing input to the stereoscopic principlebased depth measurement with which the depthmap originated from the atleast one TOF camera may be fused.

In another derived embodiment of the present invention usingstereoscopic technique, at least two TOF cameras are driven each withdifferent parameters for providing two depthmaps of the same scene withdifferent intrinsic measurement quality. Those depthmaps are fusedtogether for providing a higher quality depthmap than anyone of the twooriginal individual depthmaps. The TOF camera system may further use thetwo individual IR illumination or confidence maps natively provided bythe two TOF cameras so has to implement a stereoscopic techniquegenerating a depthmap from stereo which may be used for fusing andrefining at least one of the two depthmaps from the TOF cameras, or thedepthmap generated by their fusion. Such an embodiment may particularlybe relevant for obtaining, for instance, extra distance measurementrange that the predetermined light pulse frequencies or the illuminationpower do not allow to obtain.

In one particular embodiment wherein at least one of the sensors is aTOF sensor for being operated with respect to the TOF principle, atleast two other sensors may be RGB sensors operated with differentparameters, having a higher resolution and being used for determining adepthmap from stereovision principle. This stereovision based highresolution depthmap may be used for fusion with the lower resolutiondepthmap obtained from the TOF principle on the at least one TOF sensor.Stereovision based depthmap suffering from holes and lowest depthestimation than a TOF principle depth measurement, the depthmap obtainedat the TOF camera may be used to refine the higher resolution butuncompleted depthmap obtained by stereovision principle. Preferably thefusion may be operated within the circuitry of the TOF camera system,and the resulting improved depthmap may also comprise colour informationoriginated from the stereovision capture. This improved resulting imagebeing of a resolution at least similar to the one of the highly resolvedsensor, but may also be of a lower or higher resolution usinginterpolation computation means from state of the art.

According to another embodiment, another driving parameter that may beimplemented on the cameras of the TOF camera system, and in particularon the TOF cameras of the TOF camera system, is the use of differentfrequencies applied to the emitted pulsed illumination and theirsynchronized captures when impinging back from the scene onto eachindividual TOF camera. This particular embodiment for drivingdifferently the cameras is intended to apply depth measurementdealiasinq principle on the TOF measurements. In signal processing andrelated disciplines, aliasing refers to an effect that causes differentsignals to become indistinguishable when sampled. Temporal aliasing iswhen the samples become indistinguishable in time. Temporal aliasing canoccur when the signal being sampled periodically also has periodiccontent. In TOF principle operated systems, at a given modulationfrequency, depth aliasing results in ambiguity concerning the distanceto be recorded as same distance may be measured for object being atdifferent distances from the TOF camera system that have a predeterminedoperating range. For instance, a TOF camera system operated with asingle modulation frequency having an operating range from one meter tofive meters, makes any object at six meter from the camera system beingmeasured as being at one meter (periodic behavior), if reflecting backenough the modulated light onto the camera.

In one embodiment, at least one of the TOF cameras of the TOF camerasystem may be driven by such a dealiasing principle and moreparticularly by the related dealiasing algorithm or method. This atleast one TOF camera may be operated and driven for measuring distanceinformation according to the TOF principle using at least two differentfrequencies and the distance measurement obtained by this TOF camera maybe dealiased according to the dealiazing principle. The distancemeasurements, in the form of a depthmap, may then be fused with measuredinformation from the other cameras of the TOF camera system, said othercameras being driven with different parameters. For instance, the otherinformation may be at least one of a higher or a lower resolutiondepthmap originated from stereovision principle or from TOF principle,and/or a colour image.

In a further preferred embodiment, different dealiasing techniques maybe implemented for the different cameras, i.e. the regions 107, yieldingeven more robust dealiasing advantages as each camera provides differentdealiased depth measurements. Another example is a TOF camera systemcomprising at least two TOF cameras operated with different parameters,said different parameters being the modulation frequency to which theirrespective capture is synchronized to. At least two differentfrequencies can be used to drive the TOF cameras. The modulatedilluminating light may comprise at least two predetermined frequencies,one reference frequency and a further frequency being for instance threetimes lower than the reference frequency. One first TOF camera of theTOF camera system may be driven in synchrony with the three times lowermodulation frequency while the other TOF camera of the TOF camera systemmay be driven in synchrony with the reference frequency. This way, thetwo TOF cameras of the TOF camera system may acquire within the sametime depth aliased measurements with different unambiguous distancerange, those depth measurements may further be combined for providingone single dealiased depthmap. This principle can be repeated if needed,hence yielding a very high unambiguous distance to the complete TOFcamera system

In one derived embodiment comprising at least one TOF camera operatedaccording to the TOF principle, the dealiazed depthmap so generated mayfurther be fused with other measurements from at least one other camera,said other measurement being at least one of another same resolutiondepthmap originated from TOF principle or stereovision principle, a sameresolution colour map, a higher resolution depthmap originated from TOFprinciple or stereovision principle, a higher resolution resolutioncolour map.

It is to be noted that when using a plurality of frequencies, i.e. atleast two, for operating dealiazing principle on TOF based depthmeasurements, the higher the second frequency, the higher the accuracyof that second depth measurement. By the way, if a TOF camera systemcomprising at least one TOF camera is operated according to thedealiazing principle, and preferably if two TOF camera are beingoperated each with at least one frequency, then the fusion of the depthmeasurements may lead to a more accurate depthmap. If additionally atleast one of the cameras operated with another driving parameter is ofhigher resolution, the resulting image will comprise higher resolution,higher accuracy, and dealiased depth measurements. Even more preferably,the camera system may further comprise means for capturing colorinformation, those means being characterized in that at least one of thecameras captures colour information. Even more preferably, at least oneof the cameras of the TOF camera system is a RGBZ camera such as a RGBZsensor. The TOF camera system can thus comprise at least three cameras,at least two of the cameras being TOF cameras, the at least two TOFcameras being driven by different driving parameters, such as, but notlimited to frequencies, while imaging simultaneously the same scene.

In one further embodiment, different background light robustnessmechanisms may be implemented on the cameras. Quite often, by improvingbackground light robustness, noise or pixel pitch can be increased. Theuse of background light robustness mechanisms on different regions 107i.e. on cameras may confer strong advantages. In one embodiment, atleast one of the cameras of the system may be driven by a backgroundlight robustness mechanism. This can have advantages for applicationswhere only the resolution of one region 107 is needed in case of highbackground light.

In one further embodiment, at least two cameras of the TOF camera systemmay be driven with two different integration times. Indeed, a very shortintegration time yields high motion robustness, but also high standarddeviations on the depth values, referred to in this document as depthnoise. Therefore, a region 107 may be optimized for short integrationtime while another region 107 may be optimized for noise performance.When fusing the images and more particularly their associatedinformation, the advantages of both configurations may be obtained andused. Advantageously, this embodiment enables each fused pixel to getreliable information about fast moving objects thanks to the TOF cameradriven by a short integration time, while inheriting low noiseinformation from the others cameras driven by longer integration times.In a derived embodiment, the other cameras may comprise at least oneanother TOF camera driven with a longer integration time. In one anotherembodiment, the other cameras may comprise at least another TOF cameradriven with a longer integration time and at least one colour camera.

In order to proceed with a reliable fusion of the different information,process is to be implemented, in the circuitry, or in a companion chip,or onto a separated processing unit so as to transform the differentsets of information associated each with a coordinate system into onesingle set of data having a single common predetermined coordinatesystem. Preferably, the common predetermined coordinate system will bethe x-y plan (e.g. the plan defined by horizontal and vertical axis) ofone of the cameras, for instance the x-y plan of the highly resolvedcamera. The data from the other camera, for instance the colour images,the depthmap measurements or the greyscale image of a TOF confidencemap, are projected using the registration into an image associated withthe common predetermined coordinate system. In particular, imageregistration here involves spatially registering a target image, forinstance a low resolution highly accurate depthmap obtained form a TOFmeasurement to align with a reference image, for instance a highresolution low accuracy depthmap obtained from stereovision andcomprising colour information. Several methods of images registrationmay be used such as intensity-based or feature-based methods.Intensity-based methods may in particular compare intensity patterns inimages via a correlation metrics, while feature-based methods mostlytries to find a matching or correspondence between image features suchas points, lines, contours and depth. Intensity-based methods aim atregistering entire images or sub-images. If sub-images are registered,centres of corresponding sub-images are treated as corresponding featurepoints. Feature-based methods establish a correspondence between apredetermined number of distinctive points in images. Knowing thecorrespondence between a number of points in images, a transformation isthen determined to map the target image to the reference images, therebyestablishing point-by-point correspondence between the reference andtarget images. This later registration process may further includeinterpolation technics as images may be of different resolution.

In one preferred embodiment of the invention using image registrationwhen multiple TOF cameras are used, or at least when the TOF camerasystem comprises at least one camera providing depth information, thedepth information may be used to facilitate the fusion of the images.Depth is a unique characteristic of a scene, in first order independentof angle of viewpoint and/or light conditions. Therefore this is a verystable metric for performing any alignment, any pattern recognition orany other means needed in fusing the images.

In one preferred embodiment, at least one of the cameras could becalibrated more thoroughly, allowing the other cameras to inherit fromthis calibration. In Time-of-Flight imaging, thorough calibration stepsare required, such as absolute distance calibration, temperature,deformations, multi-path resolving and more. Calibrating only one camerasaves time due to the fewer pixels and higher mathematics that can beapplied to compute the calibration, the other cameras can then benefitand inherit the calibrated viewpoint to correct for distance errorsand/or non-linearities. This calibration may be performed at productiontime, but may also be executed at run-time, by for instance in one ofthe above mentioned TOF camera system comprising four TOF cameras,dimensioning at least one of the four viewpoints/cameras to be a muchmore stable imager, so that it is used as the reference for calibrating.

According to a further embodiment of the invention, the TOF camerasystem may further comprise means for filtering the light in the visiblerange and/or in the InfraRed. Colour filters may be implemented on topof cameras, as shown in FIG. 6. In this Figure, R, G, B and IR areasstand for Red, Green, Blue and InfraRed pass filters, respectively. Thisallows combining both RGB and depth data in one image, allowing for afused or improved image combining all these properties. However, a TOFcamera system comprising at least one TOF camera, and at least oneanother camera driven with different parameter may be characterized inthat at least one of the cameras is a RGBZ camera. A RGBZ camera is acamera comprising several pixels characterized in that the sensing areasof said pixels collect at least one colour among the Red, the Green, theBlue, preferably the three RGB colours, and additionally captureInfra-Red illumination from which a depth (Z) information may beprocessed with respect to, for instance, the TOF principle.

In another further embodiment, the pixels of at least one camera of theTOF camera system may further comprise a Quantum Dots films. QuantumDots are nanoparticles of semiconductor materials, with a diameter rangefrom 2 to 10 nm. Quantum dots demonstrate unique optical and electricalproperties due to their small size; i.e. their properties are differentin character to those of the corresponding bulk material. The mainapparent property is the emission of photons under excitation(fluorescence), which may be visible to the human eye as light orinvisible if emitting in the Infra-Red domain. The wavelength of theemitted photons depends not only on the material from which the quantumdot is made, but also on the size of the Quantum Dot. The ability toprecisely control the size of a Quantum Dot enables the manufacturer todetermine the wavelength of the emission, i.e. to determine thewavelength of light output. Quantum dots can therefore be “tuned” duringproduction to emit any wavelength desired. The ability to control, or“tune” the emission from the quantum dot by changing its core size iscalled the “size quantisation effect”. The smaller the dot, the closerit is to the blue end of the spectrum, and the larger the dot, thecloser to the red end. Quantum Dots can even be tuned beyond visiblelight, into the infra-red or into the ultra-violet, by using somespecific materials.

Used as colour filters, the Quantum Dot films may be designed forre-emitting wavelength in the range for which the sensor is moresensitive. Preferably, the emitting wavelength of Quantum Dot films maybe close to the maximum of sensitivity of the sensor enabling ameasurement of lower noise.

The invention claimed is:
 1. A sensor system comprising: a first time-of-flight (TOF) sensor configured to detect a distance to an object; and an image sensor configured to capture an image of the object, wherein the first TOF sensor and the image sensor are disposed on a common substrate and are configured to sense the object simultaneously, and wherein the first TOF sensor is configured to be driven by first driving parameters and the image sensor is configured to be driven by second driving parameters.
 2. The sensor system according to claim 1, wherein the second driving parameters are different than the first driving parameters.
 3. The sensor system according to claim 1, wherein the first driving parameters and the second driving parameters comprise at least two different frequencies for implementing a dealiasing algorithm.
 4. The sensor system according to claim 1, further comprising an array of lenses, each lens of the array of lenses being associated with a respective sensor of the first TOF sensor and the image sensor.
 5. The sensor system according to claim 3, wherein the at least two different frequencies comprise modulation frequencies configured to control a timing of the imaging of the scene.
 6. The sensor system according to claim 3, wherein the dealiasing algorithm includes instructions to distinguish between two potential distance measurements generated by the first TOF sensor.
 7. The sensor system according to claim 1, further comprising a second TOF sensor.
 8. The sensor system according to claim 7, further comprising circuitry configured to generate a dealiased depth map by combining the distance measurements from the first TOF sensor and the second TOF sensor.
 9. The sensor system according to claim 8, wherein the circuitry is further configured to implement at least two dealiasing algorithms.
 10. The sensor system according to claim 8, wherein the circuitry is disposed on the common substrate.
 11. The sensor system according to claim 1, wherein the common substrate is a silicon substrate.
 12. The sensor system according to claim 1, wherein the image sensor includes color filters corresponding to R, G, and B colors, and the first TOF sensor further includes an infrared (IR) filter.
 13. The sensor system according to claim 1, wherein a resolution of distance information detected by the first TOF sensor is different from a resolution of the image captured by the image sensor.
 14. The sensor system according to claim 13, wherein the resolution of the image sensor is higher than the resolution of the distance information.
 15. The sensor system according to claim 1, further comprising circuitry configured to: create a fusion image of the object using the detected distance to the object and the captured image of the object; and output the fusion image.
 16. A method of operating a sensor system comprising a plurality of sensors, the method comprising: detecting a distance to an object using a time-of-flight (TOF) sensor disposed on a substrate; capturing an image of the object using an image sensor disposed on the substrate; driving the TOF sensor with first driving parameters; and driving the image sensor with second driving parameters, wherein: the TOF sensor and the image sensor are configured to sense the object simultaneously.
 17. The method of claim 16, further comprising implementing a dealiasing algorithm using the first driving parameters and the second driving parameters, the first driving parameters and the second driving parameters comprising at least two different frequencies.
 18. The method of claim 17, wherein implementing the dealiasing algorithm comprises: distinguishing between two potential distance measurements generated by the TOF sensor; and generating a dealiased depth map by combining the two potential distance measurements from the TOF sensor and distance measurements from the image sensor.
 19. The method of claim 16, wherein: detecting the distance to the object using the TOF sensor comprises detecting the distance to the object with a resolution of distance information; and capturing the image of the object using the image sensor comprises capturing the image of the object with a resolution of image information, wherein the resolution of the image information is higher than the resolution of the distance information. 